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Abstract 


The use of proper measurements of diffusion of information technology 
as an innovation are essential to determining if progress is being made 
in state, regional, and national level programs. This project provides a 
national level cross validation study of several instruments commonly 
used to assess the effectiveness of technology integration in instructional 
programs. (Keywords: measures, assessment, cross-validation, technology 
integration, instruments .) 


Introduction 

T here remains in the field of Educational Technology a serious lack 
of consensus as to what measures of technological innovation on 
the part of the teacher transfer to academic achievement on the 
part of the student. Measures of diffusion of information technology as 
an innovation that are based upon solid theoretical perspectives include 
instruments such as the Concerns-Based Adoption Model Stages of 
Concern (CBAM-SoC) (Hall & Rutherford, 1974; Hall, George, & 
Rutherford, 1986, 1998), the Concerns-Based Adoption Model Levels 
of Use (CBAM-LoU) (Hall, Loucks, Rutherford, & Newlove, 1975), the 
Apple Classrooms of Tomorrow Teacher Stages (ACOT) (Dwyer, 1994), 
the Levels of Technology Implementation (LoTi) (Moersch, 1995), and 
the Stages of Adoption of Technology (Christensen, 1997; Christensen 
& Knezek, 1999), based on Russell (1995). 

Studies of reliability and construct (factor analytic) validity have 
been carried out within some of these measurement domains, but no 
known study has cross- validated these instruments against each other on 
a national basis. It is possible that this cross validation, when examined 
with student outcomes in terms of achievement scores on a standardized 
form, such as the Iowa Test of Basic Skills, may yield insight into potential 
relationships between technology integration and student outcomes. This 
type of investigation may also inform the design of instruments, in order 
to improve measurement of integration of information technology as an 
innovation at the classroom level. 

Conceptual Rationale 

The long-range purpose of this investigation is to cross-validate measures 
of technology implementation and correlate those measures with student 
achievement as a means of establishing the potential of diffusion of innova- 
tion indices to impact learning. The project has two long-term goals: first, 
cross-validate measures of technology implementation to establish valid 
and reliable instruments for measuring technology adoption at classroom 
level by the teachers; second, identify the potential impact of the adoption 
of the technology by correlating technology adoption to student achieve- 
ment scores on standardized tests. This paper addresses goal number one. 


The significance of the study rests in the fact that roughly $496 million 1 
dollars has been invested in integrating technology at the classroom level 
in the United States with no consistent means available for researchers to 
evaluate whether this expenditure is worthwhile. Multiple instruments 
are currently being used to assess project outcomes with little or no ef- 
forts being made to cross validate these measures, or correlate results to 
student outcomes in a manner helpful to policymakers. 

Multiple measures of diffusion of information technology as an 
innovation have been used to evaluate NSF, NASA, and other federal 
projects as well as statewide initiatives in education. However, there 
still remain some unresolved issues with these measures. No broad-scale 
study has attempted to cross-validate these instruments and examine 
their predictive value in terms of student achievement as measured by a 
nationally recognized achievement test. It is the purpose of the review of 
literature in the next section to discuss how these tests have been used to 
assess outcomes of large-scale initiatives, and to provide an overview of 
the instruments themselves. 

Literature Review 

Each of the selected instruments was chosen because it has been used 
for evaluation of NSF, NASA, or other large-scale implementations of 
educational initiatives. The Concerns-Based Adoption Model Stages 
of Concern (CBAM-SoC) (Hall & Rutherford 1974; Hall, George, & 
Rutherford, 1998), a basis for several of the later instruments, was used 
in Project CATS (Coordinated and Thematic Science), a National Science 
Foundation program for science in West Virginia, Rock Camp, a geology 
program with the West Virginia Bureau of Economic Geology; and in 
Project MERIT, a math initiative of the West Virginia Department of 
Education. Figure 1 (page 16) indicates results of a West Virgina study 
using CBAM-SoC as a means of assessing concerns regarding statewide 
implementation of an information management system. 

CBAM-SoC has also been used in the evaluation of educational pro- 
grams for the National Aeronautics and Space Administration (NASA) 
(Dreschel, 1996). Table 1 (page 16) indicates an alignment of NASA 
program goals and the CBAM-SoC. 

CBAM-SoC is not alone in being used to evaluate large-scale initia- 
tives. CBAM-LoU and Stages of Adoption are often used concurrently 
to assess project outcomes. For example, Figures 2 and 3 (page 17) show 
findings from a Central Nevada Technology Consortium report (Velas- 
quez-Bryant & Shonkwiler, 2004) that employed both instruments. 

Velasquez-Bryant & Shonkwiler (2004) also pointed out that the 
lack of well-publicized information regarding potential validity faults in 
the instruments makes their use for evaluation purposes more difficult. 
Interpretation of findings is more straightforward when this information 


1 Enhancing Education Through Technology (EETT) program (Title II part D of NCLB) funding level in FY05. 
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Figure 1: Normed percentiles presented as part of an analysis of a MIS in 
West Virginia. 


exists. For example, in Figure 1 note the normed percentiles for Stage 
Four of CBAM-SoC. In the 1 974-1976 study, the participants’ concerns 
were estimated after individual interviews. The estimates of their peak 
stages of concern were compared to their actual peak stages measured by 
the SoCQ. The correlation of the peak stage estimates by investigators to 
actual percentile scores shows for all stages, except Stage 4, that validity was 
supported (Table 2). These validity studies provided increased confidence 
that the SoCQ measures the hypothesized Stages of Concern in all areas 
except Stage 4 (Hall, George, & Rutherford, 1986). 

Reliability and validity issues have not often been addressed and dis- 
seminated for recent broad scale studies using these instruments, even 
as they continue to be used for evaluating interventions. Large-scale 
analyses are important so that researchers and program directors who 
apply these instruments make policy decisions based upon conclusions 
that are well grounded. 

History of Instrumentation 

The Concerns-Based Adoption Model (CBAM) was designed for the study 
of the adoption of any new educational innovation (Hall & Rutherford, 
1974). CBAM Stages of Concern (1974) and Level of Use (Hall, Loucks, 
Rutherford, & Newlove, 1975) classifications were later constructed 
to address information technology innovations and have been used by 
researchers worldwide over the past three decades. CBAM measures 
concerns toward change. In 1979, Hall reported that at the beginning of 
an innovation, individuals have high levels of concern at Stages O-Aware- 
ness, Stage 1 -Informational, and Stage 2-Personal. First, individuals are 
concerned about becoming more knowledgeable about the innovation 
and how it affects them personally. As individuals use the innovation and 
become more knowledgeable, then Stage 3-Management concerns are 
more evident. With more experience and skill, the individual’s concerns 
with the innovation move toward Stage 4-Consequence, Stage 5-Col- 
laboration, and Stage 6-Refocusing. Ownership of the innovation and 
commitment to its use are reflected in the higher stages. 

Over time, CBAM as a concept evolved into CBAM-SoCQ, referred 
to as Stages of Concern. The survey was developed around the theory 
that individual concerns about an innovation move through stages that 
can be identified. The stages mentioned previously reflect a level of in- 
tensity based upon the respondent’s feelings and perceptions about the 
innovation. Hall, George, and Rutherford (1986) detailed the process 
of determining the internal reliability of the SoCQ as an instrument. 
In studies from 1972-1976, the SoCQ was used in cross sectional and 
longitudinal studies with 1 1 different innovations. Beginning in 1974, a 
two-year study of 830 teachers and professors provided data to calculate 


Table 1: Alignment of NASA Program Goals and the CBAM-SoC 

The Stages of Concern (from Table 1 ) and the NASA Education Goals 
and Outcome Indicators (from Figure 1) 

Stages of Concern 

NEP0 Indicators 

NEP0 Goals 

0) Awareness 

Teacher awareness and 
participation in continuing 
ed. activities 

Dissemination of information 
(NASA, 1992). 

1) Informational 

Change in teacher math 
and science knowledge. 

Increased teacher content 
knowledge (math and 
science). 

3) Management 

Changes in teacher 
attitudes and practice. 
(Lesson modification or 
enhancement) 

Increased teacher capability 
to design/implement 
stimulating & engaging 
lessons/experiences. 

4) Consequence 

Increasing student 
achievement in math/ 
science as perceived by 
the teacher. 

Increased student interest 
and achievement in math/ 
science. 

5) Collaboration 

"Multiplier" effect on 
otherteachers. 

Extend benefits to 
colleagues or participants. 

6) Refocusing 

Changes in teacher 
attitude and practice 
(Lesson Plan redesign). 

Increased teacher capability 
to design/implement more 
stimulating/engaging 
lessons/experiences. 


Table 2: Correlation of Peak Stage Estimates and Rank Order of SoC 
Percentile Scores 

N=65 Critical r = .25 


Peak SoC 
Stages 0-6 



0 

1 

2 

3 

4 

5 

6 

0 

.27 

CO 

-.11 

.02 

-.22 

-.22 

-.13 

1 

.15 

.47 

.47 

-.09 

-.11 

-.50 

-.45 

2 

.03 

CO 

CO 

.42 

-.21 

-.10 

-.24 

-.34 

3 

-.25 

-.08 

.00 

.30 

-.04 

.02 

.09 

4 

-.05 

-.22 

-.26 

-.01 

.13 

.08 

.33 

5 

-.20 

-.48 

-.20 

-.03 

.31 

.54 

.15 

6 

-.20 

-.20 

.16 

-.15 

.24 

.17 

.31 


Table 3: Degree of Reliability Among Items on SoCQ in Terms of 
Overlapping Variance 

Coefficients of Internal Reliability for the Stages of Concern Questionnaire 
N = 830 

Stages 0 1 2 3 4 5 6 

Alphas .64 .78 ..83 .75 .76 .82 .71 


the coefficients of internal reliability found in Table 3. The alpha coef- 
ficients reflect the degree of reliability among items on a scale in terms of 
overlapping variance (Hall, George, & Rutherford, 1986). 

Intercorrelation matrices, judgments of concerns based on interview 
data, and confirmation of expected group differences and changes over 
time were components used to investigate the validity of questionnaire 
scores. The intercorrelation of the stages conducted by Hall and others 
is shown in Table 4. 
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Table 4: Intercorrelation of SoCQ Questionnaire Scales 


Intercorrelation of 195-Item Stages of Concern Questionnaire 



1 2 

3 

4 

5 

6 

1 

1.00 .68 

.47 

.21 

.21 

.19 

2 

1.00 

.78 

CO 

.37 

.43 

3 


1.00 

.60 

.51 

.59 

4 



1.00 

.82 

.80 

5 




1.00 

.77 


6 1.00 

The Concerns-based Adoption Model-Levels of Lise (CBAM-LoU) 
questionnaire (Griffin & Christensen, 1999 is targeted toward describing 
behaviors of innovation users through various stages-from orienting, to 
managing, and finally to integrating use of the technology. Designed to 
be more quickly administered than SoCQ, CBAM-LoU does not focus 
on attitudinal, motivational, or other affective aspects of the user. 

The instrument is based on the eight levels of use defined in the 
Levels of Use Chart (Loucks, Newlove, & Hall, 1975). The levels of 
use are: (0) Non-Use, (I) Orientation, (II) Preparation, (III) Mechani- 
cal Use, (IVA) Routine, (IVB) Refinement, (V) Integration, and (VI) 
Renewal. The concept of levels of use also applies to groups and entire 
institutions. Because the Concerns- Based Adoption Model-Levels of Use 
is a single item survey, internal consistency reliability measures cannot 
be calculated for data gathered through it. However, test-retest reliability 
estimates have been found to generally fall in the range of .84 to .87 
for elementary and secondary school teachers (Christensen, Parker, & 
Knezek, 2005, p. 189). 

The Stages of Adoption of Technology instrument is a single-item 
survey used in both preservice and inservice education to measure the 
impact of information technology training as well as trends over time. 
It was derived from work of Russell (1995) in research assessing adults 
learning to use electronic mail. Russell’s stages included: (1) awareness, 
(2) learning the process, (3) understanding the application of the process, 
(4) familiarity and confidence, (5) adaptation to other contexts, and (6) 
creative applications to new contexts. In the Stages of Adoption of Tech- 
nology instrument (Christensen, 1997; Christensen & Knezek, 1999) 
the stage descriptions are generalized to make them appropriate for any 
information technology. 

As with the CBAM-LoU, the Stages of Adoption of Technology instru- 
ment is a single item survey, internal consistency reliability measures can- 
not be calculated for data gathered through it. However, high test-retest 
reliability estimates (.9 1-.96) have been obtained from validation studies 
on Stages of Adoption (Christensen, Parker, & Knezek, 2005, p. 189). 

By the time ACOT research ended in 1998, Apple Classrooms of 
Tomorrow, had been working with the National Science Foundation 
for more than eight years on professional development for teachers in 
different environments. In its entirety, the ACOT project was one of the 
largest and longest continuing educational studies of its kind. 

Drs. Eva Baker, Joan Herman-Cooper, and Maryl Gearhart of the 
Center for the Study of Evaluation at UCLA designed and implemented 
a three-year, cross-site study of ACOT. Student demographic and psy- 
chometric data were collected annually from participating districts, using 
subsets of the Iowa Tests of Basic Skills and other measures. The project 
analysis (Baker, Herman, & Gearhart, 1988) indicated a progression of 
attitudinal change that the authors believed could be viewed as an evo- 
lutionary process similar to other models of educational change (Berman 
& McLaughlin, 1976; Giacquinta, 1973; Gross & Herriott, 1979) such 
as CBAM. This served as the basis for what would become labeled stages 
of instructional evolution in the ACOT classrooms: Entry, Adoption, 



Figure 2: CBAM-LoU results by county for Nevada report. 



Figure 3: Stages of Adoption results by county for Nevada report. 


Adaptation, Appropriation, and Invention (See Figure 4, page 1 8) . In this 
model, text-based curriculum delivered in a lecture-recitation-seatwork 
mode is first strengthened through the use of technology, and then gradu- 
ally replaced by far more dynamic learning experiences for the students 
(Dwyer, Ringstaff, & Sandholtz, 1991). 

Here again we see a pattern of a five to eight item scale emerging as 
a way of gauging progress in educational initiatives, yet there is little in- 
formation at all in the publicly available (on the ACOT Web site) reports 
that deals with reliability or validity of the ACOT stages of instructional 
evolution. This is also true of any information regarding cross-validation 
of the ACOT stages with other instruments. The five-stage version of 
the ACOT used in this study is based upon an adaptation by Tom Clark 
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PHASE 

INSTRUCTIONAL 

TECHNOLOGY 

PEDAGOGY 

OUTCOME 

Entry 

Text 

Lecture 

Recitation 

Seatworfc 

I _ r i : 

Adoption 

Text 

Lecture 

Recitation 

Seatworfc 

Soaa & Cognitive 


High 

Computer 

Access 



Adaptation 

Text 

Lecture 

Recitation 

Seatworfc 

Soaa & Cognitive 


High 

Psay & Experiment 

:• ; . ' it , 


Computer 

Access 



Appropriation 

Text 

Lecture 

Recitation 

Seatworfc 

Soaa & Cognitive 


High 

Computer 

Access 

Individuated 

Cooperative 

Project-based 

Smulaton 

Interdiscipline 

Distance 

Multmoda 

Self-paced 

Soaa & Cognitive 

Invention 

Immediate 

Computer 

Access 

Interact 

Do 

Create 

Social & Cognitive 


Figure 4.AC0T Stages of Evolution with correlating pedagogy and outcomes. 


of Western Illinois University for evaluation of a U.S. Department of 
Education STAR Schools project (Clark, 2002). 

Each year the United States and other countries spend billions of 
dollars on technology in education. A great many of these programs 
are using these instruments to evaluate program outcomes, yet the in- 
struments have not received the rigorous attention necessary to render 
them necessarily fit for evaluatory purposes. It is effort well invested to 
examine these instruments thoroughly enough to ensure that if they are 
being used to evaluate technology- assisted programs in education, we are 
getting accurate answers. 

Methods 

The study was conceived to proceed in two phases: 

1. Phase I: Cross Validation of Instruments to measure technology 
integration 

2. Phase II: Impact of Technology Integration in Learning 

The results presented here are the end of the first phase of analysis. 
Results from the second phase are not yet complete and will be reported 
in a separate publication. 

Phase I: Cross Validation of Instruments to measure 
technology integration 

This study relied upon factor analysis and reliability analysis (Cron- 
bach’s Alpha) to cross-validate three instruments: 

• The Concerns- Based Adoption Model Levels of Use (CBAM-LoU) 
(Hall, Loucks, Rutherford, & Newlove, 1975) 

• The Apple Classrooms of Tomorrow Teacher Stages (ACOT) (Dw- 
yer, 1994) 

• The Stages of Adoption of Technology (Christensen, 1997; Chris- 
tensen & Knezek, 1999) 

Factor analysis is commonly used to assess construct validity because it 
establishes relationships between latent (unobserved) variables and mul- 
tiple observable items (Holbert & Stephenson, 2002). Latent variables are 
the underlying constructs not directly tapped by any one set of measures 
(2002), but they are hypothesized to influence certain observable items 


Table 5: Participant Representation by State 


State 

Frequency 

Percent 

Valid Percent 

Cumulative 





Percent 

Alabama 

28 

2.6 

2.6 

2.6 

Alaska 

2 

.2 

.2 

2.8 

Arizona 

24 

2.2 

2.2 

5.0 

Arkansas 

9 

.8 

.8 

5.8 

California 

70 

6.5 

6.5 

12.3 

Colorado 

7 

.6 

.6 

13.0 

Connecticut 

22 

2.0 

2.0 

15.0 

Delawa re 

4 

.4 

.4 

15.4 

Florida 

67 

6.2 

6.2 

21.6 

Georgia 

41 

CO 

CO 

3.8 

25.4 

Hawaii 

8 

.7 

.7 

26.2 

Idaho 

9 

.8 

.8 

27.0 

Illinois 

38 

3.5 

3.5 

30.5 

Indiana 

23 

2.1 

2.1 

32.7 

Iowa 

3 

.3 

.3 

32.9 

Kansas 

13 

1.2 

1.2 

34.1 

Kentucky 

25 

2.3 

2.3 

36.5 

Louisiana 

24 

2.2 

2.2 

38.7 

Maine 

3 

.3 

.3 

39.0 

Maryland 

28 

2.6 

2.6 

41.6 

Massachusetts 

25 

2.3 

2.3 

43.9 

Michigan 

40 

3.7 

3.7 

47.6 

Minnesota 

10 

.9 

.9 

48.5 

Mississippi 

10 

.9 

.9 

49.4 

Missouri 

17 

1.6 

1.6 

51.0 

Montana&Nebraska 

8 

.7 

.7 

51.8 

Nevada 

6 

.6 

.6 

52.3 

New Hampshire 

2 

.2 

.2 

52.5 

New Jersey 

43 

4.0 

4.0 

56.5 

New Mexico 

11 

1.0 

1.0 

57.5 

New York 

66 

6.1 

6.1 

63.6 

North Carolina 

33 

3.1 

3.1 

66.7 

North Dakota 

3 

.3 

.3 

67.0 

Ohio 

51 

4.7 

4.7 

71.7 

Oklahoma 

19 

1.8 

1.8 

73.5 

Oregon 

9 

.8 

.8 

74.3 

Pennsylvania 

80 

7.4 

7.4 

81.7 

Rhode Island 

5 

.5 

.5 

82.2 

South Carolina 

9 

.8 

.8 

83.0 

Tennessee 

21 

1.9 

1.9 

85.0 

Texas 

84 

7.8 

7.8 

92.8 

Utah 

4 

.4 

.4 

93.1 

Virginia 

26 

2.4 

2.4 

95.5 

Washington 

27 

2.5 

2.5 

98.1 

Washington DC 

6 

.6 

.6 

98.6 

West Virginia 

4 

.4 

.4 

99.0 

Wisconsin 

5 

.5 

.5 

99.4 

Wyoming 

6 

.6 

.6 

100.0 

Total 

1078 

100.0 

100.0 



in the model. The latent variables are what a researcher ultimately wishes 
to capture, but which cannot be assessed directly through any one form 
of observation (Duncan, 1975). 

Reliability analysis is used to assess the consistency of a measurement 
scale. Cronbach’s Alpha is a common index used to determine whether 
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Table 6: Factor Extraction Summary for ACOT, CBAM-Loll, and Stages of 
Adoption Data 

Total Variance Explained 



Initial 

Eigenvalues 


Extraction 
Sums of 
Squared 
Loadings 



Component 

Total 

%of 

Cumulative 

Total 

%of 

Cumulative 



Variance 

% 


Variance 

% 

1 

2.270 

75.659 

75.659 

2.270 

75.659 

75.659 

2 

.425 

14.161 

89.820 




3 

.305 

10.180 

100.000 





Extraction Method: Principal Component Analysis. 

Table 7: Factor Loadings for ACOT, CBAM-Loll, and Stages of Adoption 
on Common Construct 

Component Matrix (Factor Loadings) 

Component 

1 

STAGES .873 

CBAM .892 

ACOT .844 

Extraction Method: Principal Component Analysis. A 1 components extracted. 

Table 8: Reliability Analysis for Technology Integration Construct 
Composed of ACOT, CBAM-Loll, and Stages of Adoption Scales 

N of Statistics Mean Variance Std Dev Variables 

for SCALE 12.8583 9.2821 3.0467 3 

Item-total Statistics 

Scale Mean if Scale Variance Corrected Item Alpha if Item 

Item Deleted if Item Deleted Total Correlation Deleted 

STAGES 7.7545 4.4979 .7096 .7674 

CBAM 8.4572 3.9779 .7408 .7379 

ACOT 9.5048 4.9009 .6618 .8137 

Reliability Coefficients N of Cases = 829.0 N of Items = 3 Alpha = .8383 

multiple indicators of a construct are internally consistent with each 
other. 

Sample 

An online survey was sent out to more than 20,000 teachers and admin- 
istrators across the nation. In order to access these educators, the project 
has received support from HotChalk, a non-governmental organization 
that provided free online learning environments to secondary schools, 
supported by corporate sponsorships and advertisements. The survey 
was completed by 1,114 educators of which 51 respondents were ad- 
ministrators and 1,063 were teachers. This response comprised 5.6% of 
the accessible population. Respondents represented 47 of the 50 U.S. 
states, plus the District of Columbia. The distribution by state/location 
is shown in Table 5. 

Analysis 

A factor analysis (Principal components, varimax rotation) was conducted 
on the data to assess construct validity of the ACOT, CBAM-LoU, and 
Stages scales as collective measures of technology integration. Exploratory 
factor analysis produced a single factor with eigenvalue > 1 for the Stages, 
CBAM, and ACOT data. As shown in Table 6, the second and third fac- 
tors that theoretically could be extracted from the data had eigenvalues 


Table 9: Guidelines for Internal Consistency Measures 


Below .60 

Unacceptable 

Between .60 and .65 

Undesirable 

Between .65 and .70 

Minimally acceptable 

Between .70 and .80 

Respectable 

Between .80 and .90 

Very good 

Much above .90 

Consider shortening the scale 


(DeVellis, 1991, p. 85). 

Table 10: Pearson Correlations Among ACOT, CBAM-LoU, 
and Stages of Adoption 

STAGES CBAM 

STAGES 1 .689 

CBAM .689 1 

ACOT .585 .632 

Correlation is significant at the U.U1 level 1 2-tailedl . 

well below 1.0, and the first factor alone (with eigenvalue 2.27) accounted 
for 76% of the variance in the data. Because the eigenvalue >1.0 cutoff 
is the point at which an extracted factor accounts for more variance than 
any single variable used alone, it was judged to be a reasonable criterion 
for a study where the goal was to identify a construct that subsumes several 
variables as indicators. The underlying construct represented by factor 1 
was tentatively named technology integration by the authors. 

As shown in Table 7, factor loadings for the component matrix revealed 
that CBAM-LoU and Stages of Adoption were most strongly aligned with 
technology integration and CBAM was slightly stronger than Stages. 
Nevertheless, since the factor loadings represent the correlations of the 
measures with the construct, these values indicate all three of the measures 
are well aligned with the construct technology integration. 

Reliability Analysis 

A reliability analysis was also carried out on the data to confirm the consis- 
tency of a technology integration scale composed of ACOT, CBAM-LoU, 
and Stages of Adoption indicators. As shown in Table 8, the resulting 
Cronbach’s Alpha for the three-item scale was r = .84. 

This is in the range of Very good’ according to guidelines for internal 
consistency reliability provided by DeVellis (1991) in Table 9. 

Associations Among the Measures 

As shown in Table 10, the Pearson Product-Moment correlations among 
the three measures of technology integration were in the range of .59 
to .69, indicating roughly 35-50% of the variance is shared among the 
pairs of the three variables. Correlations among Stages, CBAM-LoU, and 
ACOT are about the same as those found in previous studies (Christensen 
& Knezek, 2001a). For example, in a 1999 North Texas study involving 
525 educators, the Pearson product-moment correlation between Stages 
of Adoption and CBAM-LoU was r = .64 (Christensen & Knezek, 2001a, 
p. 12). This indicates the 2007 sample gathered on a broad, nationwide 
basis, yields results comparable to earlier studies utilizing exclusively 
Texas (Knezek & Christensen, 2000) or Nevada (Velasquez-Bryant & 
Shonkwiler, 2004) data. 

Discrimination Ability of the Measures 

Participants providing data for this study also completed the Teachers At- 
titudes toward Computers Questionnaire (TAC; Christensen & Knezek, 
1996) as an external validation measure. Previous studies have found a 
relatively smooth progression between Stages of Adoption of Technology 
and more positive attitudes toward computers (Christensen & Knezek 


ACOT 

.585 

.632 

1 


Volume 24 / Number 1 Fall 2007 Journal of Computing in Teacher Education 19 

Copyright© 2007 ISTE (International Society for Technology in Education), 800.336.5191 (U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org 



Computer Interest by Stage of Adoption 



Stage of Adoption of Technology 


Figure 5: Trend line for Computer Interest by Stage of Adoption of Technology. 

2001b, 2001c). As shown in Figure 4, this relationship also held true 
for the nationwide sample of data for this study. There were significant 
differences by Stage of Adoption of Technology in the level of Interest 
(TAC Factor 1) exhibited by the participants (f = 18.4, 5 x 824 df, p < 
.0005). On a scale 1 (lowest) to 5 (highest), average interest by stage 
varied from 3.69 for Stage 1 educators, to 4.59 for Stage 6 educators in 
this study. Trends similar to Figure 5 were also found for CBAM-LoU, 
ACOT, and the composite measure Technology Integration produced 
from the factor scores for the Stages, CBAM, and ACOT data. All 
measures yielded highly significant (p < .0005) attitude differentiation 
across rating categories. 

As shown in Figure 6, there was a similar trend for the association 
between higher Stages of Adoption of Technology and computer anxiety. 
In this case the mean level of anxiety generally declined as the average 
Stage of Adoption increased. On a scale of 1 = least anxious to 5 = most 
anxious, Stage 1 educators in this study gave an average self-report rating 
of 2.79 on computer anxiety, while Stage 6 teachers reported an aver- 
age rating of just 1.23 on the five-point scale. Trends similar to Figure 
6 were also found for CBAM-LoU, ACOT, and the composite measure 
Technology Integration produced from the factor scores for the Stages, 
CBAM, and ACOT data. All measures yielded highly significant (p < 
.0005) computer anxiety differentiation across rating categories. 

Summary of Findings 

These findings collectively provide evidence that the three measures 
featured in this study, when used in combination, form a reliable, valid, 
measure of the construct Technology Integration that is stable across 
geographic location and time. Stages of Adoption of Techno logy, CBAM 
Level of Use, and ACOT stages of instructional evolution form a con- 
sistent self-report measure that has stable construct validity and aligns 
well with anticipated changes in educator attitudes as level of technology 
integration progresses. 

Discussion 

One practical issue related to using the three proposed indicators ofTech- 
nology Integration is how to combine the three measures into a single 
scale. Morales (2006) found that transforming two of the three indicators 
to match the scale of the third increased the precision of the composite 
measure somewhat. However, he also found that simply averaging the 
5-point ACOT, 6-point Stages, and the 8-point CBAM-LoU scale values 
for each person produced acceptable precision for the Technology Integra- 


Computer Anxiety by Stage of Adoption 



Stage of Adoption of Technology 


Figure 6: Trend line for Computer Anxiety (Lack of Comfort) by Stage of Adoption 
of Technology. 

tion construct in his Texas-Mexico cross-cultural study. A simple average 
(geometric mean) is recommended for future research. 

A second practical issue has to do with the time required for adminis- 
tration of multiple technology integration measures. The three measures 
chosen for this study were selected in part because each instrument typi- 
cally requires only 2-3 minutes to complete. The combination offers a 
quick and easy self-appraisal of level of technology integration at the 
global level. Copies of the instruments are available online at http://iittl. 
unt.edu. 

Not included in this nationwide sample of data were two other mea- 
sures of technology integration that have been widely used. These are 
the Concerns-Based Adoption Model Stages of Concern Questionnaire 
(CBAM-SoC) (Hall & Rutherford 1974; Hall, George, & Rutherford, 
1998) and the Levels of Technology Implementation Questionnaire 
(LoTi) (Moersch, 1995). Both were previously described in the literature 
review section. Further research would be required to determine if these 
or other measures would align as well as the three used in this study. 

Conclusions 

The goal of phase 1 of this project was to cross-validate the measures of 
ACOT, CBAM-LoU and Stages of Adoption against each other. Factor 
analysis and reliability analysis have confirmed high construct validity 
and good reliability (.84) for the resulting technology integration scale. 
The next step is to determine whether this three-legged construct proves 
to be capable of discriminating among groups known to differ on at- 
tributes related to technology and education (ex: online vs. face-to-face 
learning). After this is confirmed, the important phase 2 task of using 
these to predict student achievement can begin. 

It is this relationship between integration and achievement that the 
authors believe may determine the fate of educational technology. It is 
imperative that the picture we reveal of this relationship be accurate and 
above reproach. Knowing that we have multiple quick, easy, valid and 
reliable instruments for assessment is a first step in the right direction. 
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